fix(clients): replay rate-limited requests instead of surfacing the 429 - #201
Merged
Conversation
A bulk backfill — a year of history loaded through per-event writes — runs at the endpoint category budget for minutes at a time, and every rejection came back to the caller as an ordinary failure. A demo tenant load lost 25 of 383 events to this. The 429 is raised by a request dependency before the endpoint handler runs, so the rejected call had no effect and replaying it is safe even for a POST carrying no idempotency key. Nothing else is retried, since nothing else carries that guarantee. RetryingClient subclasses httpx.Client rather than installing a custom transport: httpx disables environment proxy detection whenever transport= is supplied, so a transport-level retry would quietly break proxied callers. It is installed through the public set_httpx_client hook, leaving the generated api/ layer untouched. Retry-After is applied as a ceiling on the backoff rather than as the sleep itself. The limiter is a sliding window, so the header reports the whole window — the worst case for a caller that filled its budget instantaneously, and minutes of idling for one that merely ran at the sustained rate. max_retries / retry_delay on RoboSystemsClientConfig now reach ordinary HTTP calls; previously they only fed SSE reconnects.
jfrench9
added a commit
to RoboFinSystems/robosystems-typescript-client
that referenced
this pull request
Aug 31, 2026
TypeScript counterpart of RoboFinSystems/robosystems-python-client#201. A bulk load runs at the endpoint category budget for minutes at a time, and every rejection came back to the caller as an ordinary failure. The 429 is raised by a request dependency before the endpoint handler runs, so the rejected call had no effect and replaying it is safe even for a POST carrying no idempotency key. Nothing else is retried, since nothing else carries that guarantee. createRetryingFetch composes over an existing fetch rather than replacing it, so the GraphQL client's per-request timeout wrapper stays in place and each replay gets its own full timeout. Facade REST calls go through the generated ops, which share the module-level client and take no per-call transport, so its `fetch` is the only interception point — installed from the RoboSystemsClients constructor, and never over one the caller already supplied. Retry-After is applied as a ceiling on the backoff rather than as the delay itself. The limiter is a sliding window, so the header reports the whole window — the worst case for a caller that filled its budget instantaneously, and minutes of idling for one that merely ran at the sustained rate. maxRetries / retryDelay now reach ordinary HTTP calls; previously they only fed SSE reconnects and operation polling. Set maxRetries: 0 to surface the rejection immediately, which an interactive surface may prefer over waiting.
jfrench9
added a commit
to RoboFinSystems/robosystems-typescript-client
that referenced
this pull request
Aug 31, 2026
…29 (#207) ## Summary TypeScript counterpart of RoboFinSystems/robosystems-python-client#201, for consistency across the two published SDKs. A bulk load runs at the endpoint category budget for minutes at a time, and every rejection came back to the caller as an ordinary failure — the demo tenant load that prompted the Python change lost 25 of 383 events that way, at 92% of budget with zero headroom. The API answers an exhausted budget with `429` plus `Retry-After` / `X-RateLimit-*`, and that rejection is raised by a request dependency **before** the endpoint handler runs — the rejected call had no effect, so replaying it is safe even for a `POST` carrying no idempotency key. Nothing else is retried, precisely because nothing else carries that guarantee. `maxRetries` / `retryDelay` have been on `RoboSystemsClientConfig` since the extensions clients landed but only ever fed SSE reconnects and operation polling; ordinary HTTP had no retry path. They now reach it. ## Changes Hand-written `clients/` only; generated `sdk/` and `clients/graphql/generated/` untouched. - **`clients/retry.ts`** (new) — `createRetryingFetch`, plus `backoffMs` / `retryAfterMs` and the shared constants. - **Composes over an existing fetch rather than replacing it**, so the GraphQL client's per-request timeout wrapper stays in place and each replay gets its own full timeout rather than sharing one across attempts. The global `fetch` is resolved per call, not captured, so harnesses that swap `globalThis.fetch` keep working. - **`Retry-After` is applied as a ceiling on the backoff, never as the delay.** The limiter is a sliding window, so the header reports the whole window — the worst case for a caller that filled its budget instantaneously, and minutes of idling for one that merely ran at the sustained rate. Exponential with full jitter, capped at 30s per attempt. - A `ReadableStream` body is not replayed (the first attempt consumes it), and an aborted signal stops the loop rather than sleeping through it. - **`clients/index.ts`** — the `RoboSystemsClients` constructor installs the retrying fetch on the shared generated client via `client.setConfig`. That is the only interception point available: facade REST calls go through the generated ops, which share the module-level client and take no per-call transport. **A caller who already supplied a `fetch` keeps it** — we never override an explicit choice. `createRetryingFetch` and its constants are re-exported from the package root so consumers reaching for the raw generated SDK can opt in with `client.setConfig({ fetch: createRetryingFetch() })`. - **`clients/graphql/client.ts`** — `createGraphQLClient` wraps its `timeoutFetch` in the retrying fetch, so typed reads get the same treatment as writes. New optional `maxRetries` / `retryDelay` on `GraphQLClientConfig`. - **`clients/{Ledger,Investor,Library}Client.ts`** — optional `maxRetries` / `retryDelay` on the three facade configs, threaded from `RoboSystemsClients`. - **`clients/retry.test.ts`** (new) — 15 tests: replay-until-accepted, the 429 surfacing once retries are exhausted, `maxRetries: 0` sending exactly once, other statuses untouched, a streamed body not replayed, an aborted signal short-circuiting, per-call global resolution, and the inner fetch re-running on every replay (which is what proves retry wraps timeout and not the reverse). Plus the backoff units, including that `Retry-After: 60` does not become a 60s sleep. - **`clients/index.test.ts`** — the SDK client mock gains `setConfig` (part of the real surface), and three tests cover the wiring: the retrying fetch installed, a caller-supplied fetch left alone, and the budget reaching the GraphQL facades. ## Compatibility ADDITIVE surface; one deliberate runtime-behavior change, confined to 429. - **New exports**: `createRetryingFetch`, `RetryOptions`, `DEFAULT_MAX_RETRIES`, `DEFAULT_RETRY_DELAY_MS`, `MAX_BACKOFF_MS`; optional `maxRetries` / `retryDelay` on `GraphQLClientConfig` and the three facade configs. - **Unchanged**: every facade signature and return type. No removals, no renames, no narrowed inputs, no generated-type changes. - **Behavior**: a request that would have failed on 429 now waits and retries up to `maxRetries` (default 5) before surfacing the same rejection. Nothing else is affected, and a caller who never hits a rate limit sees no change. - **Worth a look from the app owners.** The three frontends are interactive surfaces, and the default budget means a sustained rate limit can now sit behind a spinner for up to ~30s instead of erroring straight away. `maxRetries: 0` restores the old behavior and a lower value (2 → ~3s worst case) may suit a UI better than the batch-oriented default. Happy to change the default here if you'd rather it be lower for browsers. - Stable-tier signatures untouched, so this **ships as a minor**. Version bump is the release dispatch's job, not this PR's. ## Testing - `npm run test:all` (validate → test → build) in-session: prettier, eslint and `tsc --noEmit` clean, **331 passed** (18 added), build clean. - `npm run prepare:publish` confirms `artifacts/retry.{js,d.ts,ts}` lands in the published layout. - Not regenerated — no API surface change is involved, so `sdk/` was deliberately untouched.
This was referenced Aug 31, 2026
Merged
jfrench9
added a commit
that referenced
this pull request
Aug 31, 2026
…ions status/cancel calls (#202) ## Summary Two commits, both follow-ups to #201, kept in one PR because they touch the same file. **1. The 429 replay reaches the last four REST call sites.** #201 covered every facade that builds an `AuthenticatedClient` — which is what I grepped for — but `QueryClient`, `OperatorClient` and `OperationClient` resolve the credential themselves and pass it in `headers` on a plain `Client`. Cypher queries, operator runs and operation-status polls each draw on their own category budget, and polling an operation in a loop is exactly the shape that exhausts one. This also closes a divergence between the SDKs: in the TypeScript client every generated op resolves `(options.client ?? client)` against one shared singleton, so installing the retrying `fetch` covered these by construction. Python builds a client per call, so each site had to be reached individually. **2. `get_operation_status` and `cancel_operation` could not succeed at all.** Surfaced while writing a test for (1). Both read an attribute off a model carrying only `additional_properties`: ``` GetOperationStatusResponseGetoperationstatus.from_dict({"status": "completed"}).status → AttributeError: object has no attribute 'status' ``` The surrounding `except Exception` then shaped that into a plausible-looking failure for *every* response — status always returned `{"status": "error"}`, cancel always returned `False`. Both are reachable from the facade. ## Changes ### Commit 1 — retry parity - **`clients/retry.py`** — `retrying_client`, the unauthenticated sibling of `retrying_authenticated_client`. Same `set_httpx_client` install; no credential stamping, since these callers supply their own headers. - **`clients/query_client.py`** (1), **`clients/operator_client.py`** (1), **`clients/operation_client.py`** (2) — now build through it. - **`tests/test_auth_header_resolution.py`** — `test_status_call_uses_provider_credential` patched the `Client` constructor, which this routes around. It now asserts the credential on the client actually handed to the generated op — the behaviour it meant to pin, independent of how the client is built. ### Commit 2 — the operations response body - **`clients/operation_client.py`** — `_parsed_dict` reads the body through `to_dict()`, matching what `operator_client._poll_for_completion` already does, and both methods use it. - **`cancel_operation` had a second defect the first one masked.** Its SSE cleanup sat *after* an early `return` on the success path, so the one case that needs the stream closed — the cancel actually landed — was the case that skipped it. Because the AttributeError meant that `return` was never reached, the cleanup was dead code outright; fixing only the accessor would have made it permanently dead on the common path. It now runs before the outcome is returned. - **`tests/test_operation_client_status.py`** (new) — 12 tests over both methods: real status values, a failed operation's error, missing/absent body, transport failure, cancel true/false, and the stream being closed on a successful cancel. - **`tests/test_operation_client_ops.py`** — the pre-existing tests passed against all of the above because they asserted on bare `Mock`s, where attribute access always works. They now build the real response models. Verified they fail with the `AttributeError` when only `operation_client.py` is reverted. ## Deliberately excluded - **`auth_integration.py`** — the cookie-based login client. Backoff on the auth path is a security control, not a convenience. - **SSE streams** (`sse_client`, `graph_client._wait_with_sse`) — own reconnect logic. - **Presigned S3 transfers** (`file_client._http_client`, the report-bundle download in `ledger_client`) — not our API, not rate-limited by us. - **Async** — `set_httpx_client` covers the sync client only. Every facade is sync, so nothing in-tree hits it. ## Compatibility ADDITIVE surface. Two runtime-behaviour changes, both on paths that previously could not work. - **New export**: `retrying_client`. - **Unchanged**: every facade signature and return type. - **Behaviour**: a query / operator / operations call that would have failed on 429 now retries (`max_retries=0` restores the old behaviour). `get_operation_status` now returns the real status instead of `{"status": "error"}`; `cancel_operation` returns the real outcome instead of always `False`, and closes the stream when the cancel lands. **A caller that special-cased the broken shapes should be checked** — anything treating `status == "error"` as "poll again", or `cancel_operation() is False` as normal, will now see the true value. - Ships as a **minor**. Version bump is the release dispatch's job. ## Testing - `just test-all`: ruff, format, `basedpyright` 0 errors, `pytest` **583 passed / 17 skipped** (15 added, 3 rewritten). - Confirmed no plain `Client(` construction remains under `clients/` apart from the deliberate `auth_integration` one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A bulk backfill — a year of history loaded through per-event write calls — runs at the endpoint category budget for minutes at a time, and every rejection came back to the caller as an ordinary failure. A showcase demo tenant load lost 25 of 383 events to exactly this: ~500 extensions writes in ~108s against a 300/min budget, 92% of budget with zero headroom, so ordinary jitter converted straight into dropped rows.
The API answers an exhausted budget with
429plusRetry-After/X-RateLimit-*, and that rejection is raised by a request dependency before the endpoint handler runs — the rejected call had no effect, so replaying it is safe even for aPOSTcarrying no idempotency key. Nothing else is retried, precisely because nothing else carries that guarantee.max_retries/retry_delayonRoboSystemsClientConfighave existed since the extensions clients landed but only ever fed SSE reconnects; ordinary HTTP had no retry path at all. They now reach it.Changes
Hand-written facades under
robosystems_client/clients/androbosystems_client/graphql/only; generatedapi/,models/andclient.pyuntouched.clients/retry.py(new) —RetryingClient, plusbackoff_seconds/retry_after_seconds/retrying_authenticated_client.httpx.Clientrather than installing a custom transport. httpx setsallow_env_proxies = trust_env and transport is None, so a transport-level retry would silently disable environment proxy detection for every caller. Overridingsend()leaves proxy, mount, redirect and TLS handling exactly as httpx configures them.set_httpx_clienthook, sojust generate-sdkcannot wipe it.retrying_authenticated_clientreproduces whatAuthenticatedClient.get_httpx_client()would have built, including stamping the credential onto the caller's named auth header and leavingtimeoutunset to match the generated default.Retry-Afteris applied as a ceiling on the backoff, never as the sleep. The limiter is a sliding window, so the header reports the whole window — the worst case for a caller that filled its budget instantaneously, and minutes of idling for one that merely ran at the sustained rate. Backoff is exponential with full jitter, capped at 30s per attempt.hasattr(request, "_content")is the test.clients/{ledger,investor,document,graph,table,file}_client.py— all 10AuthenticatedClientconstruction sites now go throughretrying_authenticated_client(..., config=self.config). Function-scopedfrom ..client import AuthenticatedClientimports that existed only to construct it were swapped for the helper.graphql/client.py—GraphQLClienttakesmax_retries/retry_delay_msand posts throughRetryingClient, so typed reads get the same treatment as writes. Threaded fromself.configat the threeGraphQLClientconstruction sites (ledger / investor / library).tests/test_rate_limit_retry.py(new) — 10 tests against a stub server that rejects a fixed number of requests: replay-until-accepted, the 429 surfacing once retries are exhausted,max_retries=0sending exactly once, a streamed body not replayed, other statuses untouched, credential and extra headers surviving onto the retrying client, andLedgerClient.create_event_blockend-to-end through the real facade and generated op. Plus the backoff unit tests, including thatRetry-After: 60does not become a 60s sleep.Sync only.
set_httpx_clientcovers thesync_detailedpath every facade uses; a caller reaching for a generatedasyncio_detailedon one of these clients still gets a plainAsyncClientfromget_async_httpx_client().One incidental fix: the facades previously handed their
headersdict toAuthenticatedClientby reference, andget_httpx_client()mutated it in place to add the auth header — so a facade'sself.headers(and the sharedconfig["headers"]) silently grew a credential after the first REST call, which then leaked intoGraphQLClient. The helper copies instead.Compatibility
ADDITIVE surface; one deliberate runtime-behavior change, confined to 429.
robosystems_client.clients.retry(RetryingClient,retrying_authenticated_client,build_httpx_client,backoff_seconds,retry_after_seconds,RETRY_STATUS_CODES,DEFAULT_MAX_RETRIES,DEFAULT_RETRY_DELAY_MS,MAX_BACKOFF_SECONDS); two optional keyword args onGraphQLClient.__init__.max_retries(default 5) before surfacing the same error. A caller that wrapped these calls in its own 429 backoff will see the delay applied twice; settingmax_retries=0on the config restores the previous behavior exactly. No other status is affected, and a caller who never hits a rate limit sees no change.Testing
just test-allin-session:ruff checkandruff format --checkclean,basedpyright0 errors,pytest568 passed / 17 skipped (10 added).create-event-block.robosystems's own lockfile, since the demo clones that repo anduv syncs in the checkout.